Combine smaller related jobs in CI to reduce overhead#17167
Conversation
github-actions
Bot
added
delta-lake
Each individual CI job that we add has overhead of ~2m to ~4m. With almost 50 CI jobs this adds up to 100m of overhead at minimum. Combining related jobs together while still keeping the job duration under 30m helps reduce this overhead. This new combined job takes ~32m.
Each individual CI job that we add has overhead of ~2m to ~4m. With almost 50 CI jobs this adds up to 100m of overhead at minimum. Combining related jobs together while still keeping the job duration under 30m helps reduce this overhead. None of the new combined jobs take more than 30m.
hashhar
force-pushed
the
hashhar/more-ci-cleanup
branch
from
7ca0559 to
fce42ad
Compare
hashhar
force-pushed
the
hashhar/more-ci-cleanup
branch
from
fce42ad to
8362d50
Compare
|
I removed the last commit which combined |
| if: >- | ||
| matrix.modules != 'plugin/trino-singlestore' | ||
| && ! (contains(matrix.modules, 'trino-delta-lake') && contains(matrix.profile, 'cloud-tests')) | ||
| && ! (contains(matrix.modules, 'trino-delta-lake') && contains(matrix.profile, 'gcs-tests')) |
There was a problem hiding this comment.
GCS tests may have high variance in run times, since Github runners run in unspecified regions.
There was a problem hiding this comment.
All the cloud-tests run in different regions. BTW the combined runtime is much shorter than runtime of cloud-tests + gcs-tests since gcs-tests had a single test class which limited concurrent tests. So chances of crossing 30m is even lower. I'll however keep an eye on CI times for next few days for builds on master to make sure we don't regress.
4 participants
Description
Each individual CI job that we add has overhead of ~2m to ~4m. With
almost 50 CI jobs this adds up to 100m of overhead at minimum. Combining
related jobs together while still keeping the job duration under 30m
helps reduce this overhead.
Additional context and related issues
Follows #17163.
Release notes
(x) This is not user-visible or docs only and no release notes are required.